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1 .    INTRODUCTION 

The  University  of  Illinois'  computer-assisted  instruction  system, 
PLATO  (Alpert  and  Bitzer,  1970  and  Bitzer,  Sherwood,  and  Tenczar,  1972), 
currently  has  a  library  of  several  thousand  lessons.   Each  of  these  lessons 
attempts,  through  the  medium  of  an  interactive  computer  terminal,  to  present 
instructional  material  to  students. 

This  thesis  proposes  a  method  for  improving  the  effectiveness  of 
PLATO  lessons  and  describes  how  this  method  was  applied  to  a  particular 
PLATO  lesson  written  by  the  author.  The  improvement  procedure  has  three 
steps.   First,  statistics  are  gathered  about  students'  interaction  with 
the  lesson.   Secondly,  the  data  collected  is  analyzed  and  areas  in  the 
lesson  where  students  had  difficulty  are  identified.  Finally,  areas 
identified  as  deficient  are  rewritten  to  more  clearly  present  material 
the  average  student  found  puzzling. 

The  PLATO  lesson  on  which  this  improvement  technique  was  tested 
is  fortarith,  an  introductory  lesson  on  FORTRAN  arithmetic.  After  an  initial 
version  of  this  lesson  was  written  and  tested,  data  were  collected  on  the 
use  of  the  lesson  by  twenty-three  students  enrolled  in  Computer  Science  103, 
an  introductory  programming  course  at  the  University  of  Illinois.   In  this 
thesis,  the  mechanism  for  collecting  data  about  students'  interaction  with 
fortarith,  the  analysis  of  the  statistics  gathered,  and  improvements  made 
to  fortarith  as  a  result  of  the  data  collection  are  discussed. 


2.   DESIGN,  STRUCTURE,  AND  CONTENTS  OF  FORTARITH 

2 .1.   Design 

The  goal  of  fortarith  is  to  teach  a  student  how  calculations  are 
performed  in  FORTRAN  assignment  statements.  After  completing  the  lesson, 
a  student  should  be  able  to  write  assignment  statements  for  his  own  programs 
and  be  able  to  read  and  understand  assignment  statements  he  sees  in 
example  programs . 

Since  fortarith  was  designed  as  one  of  a  series  of  PLATO  lessons 
that  teach  the  entire  FORTRAN  programming  language,  certain  as sumptions  are 
made  about  the  typical  user  of  the  lesson.   The  typical  user  is  assiomed  to 
be  a  student  enrolled  in  one  of  the  basic  computer  science  courses  taught 
at  the  University  of  Illinois.   In  keeping  with  this  assumption,  fortarith 
is  designed  so  that  the  average  student  can  successfully  complete  the  lesson 
in  less  than  fifty  minutes,  the  length  of  a  regular  class  session.  And, 
since  the  typical  user  of  fortarith  will  be  running  his  programs  at  the 
University  of  Illinois,  the  particular  version  of  FORTRAN  discussed  in  the 
lesson  is  WATFIV  (Cress,  Dirksen,  and  Graham,  1970)  as  implemented  on  the 
University's  IBM  36o/75  computer. 

The  assumption  that  the  typical  user  of  fortarith  is  enrolled  in 
a  basic  computer  science  course  at  the  University  of  Illinois  has  a  minimal 
effect  on  the  contents  of  the  lesson.  Fortarith  is  general  enough  to  be 
used,  without  prerequisites,  by  anyone  with  a  high  school  knowledge  of 
mathematics  and  an  intuitive  notion  of  what  an  algorithm  is. 


2.2.  Structure 

The  structure  of  fortarith  is  similar  to  that  of  the  other  PLATO 
lessons  in  the  FORTRAN  sequence.   After  a  title  page,  the  student  is 
presented  with  a  description  of  how  the  special  function  keys  on  the  keyboard 
operate.   The  -NEXT-  key,  for  example,  is  used  to  advance  through  the  normal 
flow  of  the  lesson.  The  -BACK-  key  is  available  should  a  student  wish  to 
go  backwards  to  review  a  topic.    Pressing  shift  -NEXT-  will  jump  the 
student  forward  to  the  beginning  of  the  next  major  topic  or  subtopic.   Pressing 
shift  -BACK-  will  return  the  student  to  the  beginning  of  his  current  topic 
or  subtopic.   The  use  of  the  -TERM-  key  to  jump  to  particular  parts  of  the 
lesson  is  also  explained. 

After  the  mechanics  of  using  fortarith  have  been  covered,  the 
student  is  presented  with  a  table  of  contents  for  the  lesson  (see  Figure  1). 
From  the  table  of  contents,  the  student  can  branch  to  any  one  of  the 
twenty- two  topics  or  subtopics  listed.   After  branching  to  a  particular 
topic  or  subtopic,  the  student  proceeds  normally  through  the  lesson  by 
pressing  the  -NEXT-  key.   He  can  always  return  to  the  table  of  contents 
at  any  point  in  the  lesson  by  pressing  the  -TERM-  key  and  entering  "index". 

2.3.  Content  overview 

Fortarith  is  divided  into  six  major  topics.   In  the  first  section, 
the  idea  of  evaluating  an  expression  and  storing  the  result  in  a  memory 
location  is  introduced.  Constants,  both  integer  and  real,  are  discussed 
in  the  second  section.  The  third  section  deals  with  integer  and  real 
variable  names  and  with  type  rules.   The  fourth  section  goes  over  built-in 


Figure  1.   The  Table  of  Contents 
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functions,  while  the  fifth  section  covers  operator  precedence  rules. 

The  sixth  section  goes  over  the  differences  between  real,  integer,  and  mixed 

mode  arithmetic .  Thus,  all  aspects  of  how  calculations  are  performed 

in  FORTRAN  are  taught. 

Throughout  the  lesson,  explanatory  material  and  examples  are 
interspersed  with  various  drills.   In  general,  new  material  and  corresponding 
examples  are  presented,  followed  immediately  by  a  drill.  A  thorough  under- 
standing of  the  material  preceding  the  drill  should  enable  a  student  to 
answer  all  drill  questions  correctly. 


3-        LESSON   IMPROVEMENT 

3-1.    Some  simple  methods 

Many  PLATO  authors,  particularly  those  new  to  the  system,  make 
the  mistake  of  thinking  they  have  "finished"  a  lesson  when  they  have  written 
all  the  code  for  the  lesson's  component  parts,  tested  the  function  keys 
to  see  that  they  all  work  as  advertised,  and  corrected  all  spelling  errors 
in  the  text.  To  be  considered  "complete"  a  lesson  must,  of  course,  achieve 
such  minimal  standards.  However,  a  lesson  should  never  he  considered 
"finished"  until  it  is  shown  to  he  effective  in  teaching  the  material  it 
was  designed  to  cover. 

Taking  teaching  effectiveness  as  the  ultimate  goal  of  any 

instructional  PLATO  lesson,  the  question  immediately  arises,  how  can  the 
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effectiveness  of  a  lesson  best  be  judged.   During  the  development  of  fortarith, 

a  number  of  different  evaluation  techniques  were  applied  to  the  lesson. 
The  shortcomings  of  the  first  methods  employed  led  to  the  development  of 
the  data  collection  method  which  forms  the  basis  of  this  thesis.   However, 
before  discussing  that  method,  a  review  of  the  earlier  techniques  and 
the  reasons  why  they  proved  less  than  satisfactory  is  in  order. 

The  first  evaluation  of  fortarith  was  done  by  a  class  of  twenty-sevei 
Computer  Science  199  students  who  filled  out  questionnaires  about  the  lesson. 
(See  Appendix  A  for  a  sample  questionnaire.)  The  students  were  also 
observed  as  they  actually  took  the  lesson.   Some  valuable  comments,  both 
written  and  oral,  were  obtained  from  this  evaluation.   Indeed,  one  student 
even  found  an  erroneous  statement  in  the  text  of  the  lesson. 


However,  evaluation  by  observation  and  questionnaire  did  have 
drawbacks.  First  of  all,  close  observation  of  a  student  seems  to  have  an 
inhibiting  effect  on  his  behavior.  With  an  observer  looking  over  his 
shoulder,  a  student  tends  to  use  a  lesson  only  in  the  most  obvious, 
straightforward  manner.  An  additional  difficulty  encountered  in  this 
particular  case  was  the  impracticality  of  having  only  one  observer  (the 
author)  observe  twenty-seven  students  simultaneously. 

As  for  the  questionnaires,  the  results  were  generally  so  unspecific, 
they  were  of  little  value.   Part  of  the  problem  may  have  been  due  to  the 
type  of  questionnaire  used.  Most  questions  required  the  student  to  make  a 
check  mark  along  a  scale  of  some  sort.   Just  exactly  what  causes  a  student 
to  check  one  point  along  a  scale  instead  of  another  is  difficult  to  determine, 
especially  if  no  additional  written  comments  are  provided.  Even  a  written 
comment  like  "several  really  tricky  questions"  is  difficult  to  interpret. 
Are  "tricky  questions"  good  or  bad?  Using  teaching  effectiveness  as  the 
major  criterion,  "tricky  questions"  are  presumably  good  if  they  fix  a 
concept  more  firmly  in  a  student's  memory.   On  the  other  hand,  if  a  student 
considers  a  question  "tricky"  because  the  preceding  material  was  not 
adequately  covered,  "tricky  questions"  indicate  flaws  in  the  lesson. 
Moreover,  exactly  which  questions  were  "tricky"  and  which  were  not?   The 
data  collected  by  means  of  questionnaires  is  frustratingly  imprecise. 

As  a  second  method  of  lesson  evaluation,  professors  and  graduate 
students  with  experience  teaching  introductory  FORTRAN  courses  were  asked 
to  go  through  fortarith  and  suggest  places  where  they  felt  improvements  might 
be  made.   Sometimes  such  reviews  took  a  written  form;  at  other  times  the 
reviewer  made  his  comments  directly  to  the  author  as  they  jointly  viewed  the 

lesson.  A  number  of  excellent  suggestions  for  improvements  were  generated 
by  such  reviews . 


However,  reviewing  as  an  evaluation  technique  also  left  something 
to  he  desired.   A  review  by  its  very  nature  is  quite  subjective.   Consequently,! 
on  occasion,  there  was  disagreement  among  reviewers  as  to  which  lesson 
features  were  good  and  which  needed  improvement.  A  section  one  reviewer 
described  as  "boringly  redundant"  was  viewed  by  another  reviewer  as 
needing  further  examples. 

Another  problem  with  reviewing  as  a  technique  for  lesson  evaluation 
is  that  the  reviewers  are,  in  a  certain  sense,  too  familiar  with  FORTRAN 
and  with  PLATO.   Even  when  trying,  it  is  difficult  for  a  knowledgeable 
reviewer  to  make  the  same  sort  of  errors  a  student  unfamiliar  with  PLATO 
and  with  programming  language  s  wi 11  make . 

3.2  .   Data  collection 

The  data  collection  method  for  evaluating  lessons  is  made  possible 
by  certain  features  of  PLATO  and  of  TUTOR,  the  special  purpose  language 
in  which  PLATO  lessons  are  written.   PLATO  provides  system  "datafiles" 
as  semi -permanent  storage  where  data  can  be  collected.  Data  is  actually 
written  into  a  particular  data  file  when  special  TUTOR  output  commands 
are  executed  as  students  use  the  lesson.   Since  exactly  what  data  gets 
collected  is  determined  precisely  by  the  code  of  a  lesson,  an  author 
can  make  the  data  collection  mechanism  in  his  lesson  as  elaborate  or  as 
simple  as  he  wishes. 

The  goal  of  data  collection  is  the  generation  of  statistics  which, 
hopefully,  will  provide  some  insights  into  how  a  particular  lesson  is  being 
used  (or  misused) .   Since  the  data  collection  capability  provided  by  PLATO 
and  TUTOR  is  extremely  flexible,  an  author  is  limited  only  by  his  skill 


at  programming  as  to  what  statistics  he  can  gather  regarding  his  lesson's 
use.   Importantly,  the  effort  required  to  properly  install  data  collection 
code  need  only  be  expended  once  to  gather  an  unlimited  (theoretically, 
at  least)  amount  of  data.*  This  is  because  data  collection  is  automatic. 

Once  data  collection  code  is  built  into  a  lesson,  data  items  seemingly 
"collect  themselves"  as  data  collection  code  is  executed  by  students 
using  the  lesson. 

Thus,  data  collection  certainly  seems  to  be  a  great  improvement 
over  the  earlier  methods  of  lesson  evaluation  discussed.  Ambiguity  has  been 
totally  eliminated  by  the  automatic  and  uniform  collection  of  precisely 
defined  data  items .  Reliance  on  subjectivity,  while  not  completely 
eliminated  by  the  data  collection  method,  is  at  least  greatly  reduced.   Just 
how  effectively  a  lesson  teaches  material  is  still  a  subjective  judgment. 
However,  with  the  data  collection  method,  it  is  a  subjective  judgment  founded 
on  at  least  one  objective  basis  of  fact,  the  statistics  generated  from 
the  collected  data. 

Of  course,  no  means  of  lesson  evaluation  is  problem  free,  including 
the  data  collection  method.  None  of  the  drawbacks  to  data  collection  are 
particularly  serious,  however. 


^Realistically,  of  course,  datafiles  eventually  will  fill  up.  However, 
given  several  datafiles  and  a  way  of  emptying  a  full  datafile  onto  a 
less  critical  resource  like  magnetic  tape,  as  much  data  as  one  would 
ever  want  to  analyze  can  be  gathered. 
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The  addition  of  extra  code  to  collect  data  increases  the  size  of 
a  lesson.  Fortarith,  for  example,  increased  approximately  fifteen  percent 
in  size  due  to  the  addition  of  data  collecting  code.  A  data  collection 
mechanism  more  elaborate  and  complex  might  inflate  a  lesson's  size  even  more 
On  a  system  like  PLATO  where  disc  space  is  often  at  a  premium,  finding 
additional  lesson  space  in  which  to  add  data  collection  code  may  be  a  real 
difficulty  operationally. 

Another  problem  is  data  collection's  "invisibility."  Unless  the 
students  are  told  that  their  responses  are  being  monitored  and  stored  away 
in  a  datafile,  students  are  totally  unaware  that  they  are  under  observation. 
Slow  response  time  would  not  tip  the  student  off  because  PLATO  handles  the 
extra  work  required  for  data  collection  with  no  discernable  performance 
degradation.   One  could  view  the  effective  "invisibility"  of  data  collection 
as  a  solution  to  the  phenomenon  that  observation  inhibits  a  student  from 
using  a  lesson  in  any  but  the  most  straightforward  manner.  However,  it 
seems  highly  unethical  to  secretly  "spy"  on  students.   In  this  era  of 
post -Watergate  morality,  it  would  seem  more  proper  to  notify  students 
beginning  a  lesson  with  data  collection  that  their  responses  are  being 
recorded  as  part  of  a  lesson  improvement  technique.   Such  a  notification, 
of  course,  would  leave  unsolved  the  problem  that  observation  tends  to  be 
inhibiting. 
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k.        DETAILS  OF  FORTARITH'S  DATA  COLLECTION  MECHANISM 

Several  types  of  data  were  recorded  in  fortarith.   Perhaps 
the  most  useful  items,  in  terms  of  pointing  out  places  in  the  lesson  in 
need  of  revision,  were  the  drill  score  summary  items.   Data  of  this  variety 
summarized  students'  performance  on  a  particular  drill.   Questions  missed 
by  large  numbers  of  students  often  indicated  areas  in  the  lesson  not 
covered  thoroughly  enough. 

Wrong  answers  to  drill  questions  were  also  recorded  in  the 
dataf ile .  By  examining  the  incorrect  answers  given  by  students,  it  was 
sometimes  possible  to  identify  a  common  source  of  confusion  and  to  clarify 
a  section  of  the  lesson.   Other  data  recorded  included  the  time  spent  in 
each  of  the  major  sections  of  the  lesson  as  well  as  the  time  spent  on 
each  drill . 

Students'  use  of  the  -HELP-  function  key  and  the  -TERM-  function 
key  were  also  monitored,  the  -HELP-  key  indicating  a  desire  by  a  student 
for  further  assistance  and  the  -TERM-  key  indicating  a  desire  by  a  student 
to  either  leave  the  lesson  or  to  jump  to  another  point  within  the  lesson. 

The  TUTOR  command  most  commonly  used  to  write  data  items  into  the 
dataf ile  was  the  -outputl-  command.  This  command  outputs  a  given  number  of 
60-bit  words  to  the  dataf ile  and  attaches  a  label. 

The  individual  pieces  of  data  were  often  integers:   a  count  of 
correct  answers,  the  number  of  seconds  spent  on  a  particular  drill,  etc. 
To  conserve  space  within  the  dataf ile,  several  such  counters  were  often 
"packed"  or  compressed  into  one  word.   The  standard  scheme  used  in  fortarith, 
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which  admittedly  was  arbitrary*,  was  to  pack  five  such  integer  counters 
into  five  12 -bit  segments  forming  a  60-bit  word.   If  more  than  five 
counters  were  to  be  output  to  the  datafile,  as  many  words  as  were  necessary 
would  be  divided  into  12-bit  segments.   For  example,  after  a  student 
finished  the  variable  identification  drill,  seven  counters  were  packed  into 
two  words  and  output  to  the  datafile  with  the  label  ' varial '  (see  Figure  2). 
The  procedure  for  analyzing  the  collected  data  was  clumsy  at  best, 
in  part  due  to  limitations  in  the  PLATO  systems.  The  contents  of  the 
datafile  were  printed  as  a  listing.  From  this  printout,  cards  were 
punched  in  a  strictly  defined  format,  one  data  item  per  card.  Naturally, 
large  amounts  of  time  and  care  were  spent  verifying  that  the  data  had  been 
accurately  transcribed  from  the  listing  to  the  cards.  Eventually,  the 
carefully  checked  deck  of  approximately  800  cards  was  used  as  input  to  a 
specially  written  FORTRAN  program  which  gathered  the  statistics  which  appear 
in  the  tables  that  follow. 


^Segments  of  length  12  appeared  a  good  choice  for  two  reasons.  First  it  is 
divisible  by  three  and,  hence,  easily  displayed  in  octal.  Secondly,  it  is 
large  enough  to  hold  any  integer  up  to  20^7  • 
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Figure  2 .   Example  of  "Packed"  Counters 
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count  of  real  variables  correctly  identified 

count  of  real  variables  given  in  drill 

count  of  illegal  variables  correctly  identified 

count  of  illegal  variables  given  in  drill 

time  for  the  variable  identification  drill  (in  seconds) 

not  used 


lit 

5-   ANALYSIS  OF  THE  DATA  GATHERED  BY  FORTARTTH 

5 .1.   The  E  -conversion  drill 

The  first  major  drill  of  fortarith  asks  the  student  to  convert 
three  real  constants  given  in  exponential  form  to  simple  real  constants 
(see  Figure  3) •   Statistics  gathered  about  the  E-conversion  drill  (see 
Table  l),  led  to  two  changes  to  fortarith.  First  of  all,  twelve  of  the 
twenty-seven  incorrect  answers  to  the  drill  questions  were  null.  That  is, 

on  twelve  occasions  students  merely  pressed  the  -NEXT-  key,  were  judged  wrong, 
told  the  correct  answer,  and  allowed  to  proceed.   To  prevent  such  laziness 
on  the  part  of  students,  fortarith  was  modified  to  ignore  all  null  answers. 
In  the  new  version  of  fortarith,  students  may  only  proceed  past  a  drill 
question  by  entering  some  sort  of  non-null  response. 

Secondly,  students  commonly  forgot  the  leading  minus  sign  when 
converting  the  third  constant  -9876E-5  to  a  simple  real  constant.   Indeed, 
eight  of  the  fifteen  wrong  answers  to  the  question  were  .09876  instead  of 
-.09876.  A  special  error  message  "Oops !  You  forgot  the  minus  sign!"  was 
added  to  indicate  to  those  making  that  particular  error,  precisely  what  they 

had  done  wrong. 


Problem 

1 
2 
3 


Number  of  Incorrect  Answers 

7 

5 

15 


Number  of  Null  Answers 

It 
It 
It 


Table  1.       E-conversion  Drill  Statistics 
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Figure   3.       E -Conversion  Drill 
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5-2.   The  true -false  drill 

The  second  substantial  drill  of  fortarith  asks  the  student  to 
classify  three  statements  concerning  real  constants  as  either  true  or 
false  (see  Figure  h) .      This  drill  evidently  gave  students  little  trouble 
(see  Table  2) . 

Problem  Number  of  Incorrect  Answers 

1  1 

2  2 

3  1 

Table  2.   The  True -False  Drill 

5«3«   The  constant  identification  drill 

This  drill  (see  Figure  5)  is  the  final  exercise  in  the  section  on 
constants.  The  student  is  asked  to  identify  a  given  constant  as  integer 
in  type,  real  in  type,  or  neither  (invalid  for  one  reason  or  another).  The 
student  must  correctly  work  ten  problems  before  he  can  escape  from  the  drill, 
If  he  tries  to  leave  prematurely,  he  is  told  to  "Go  to  the  foot  of  the 
class,"  his  score  is  reset  to  zero,  and  he  is  returned  to  the  drill 
(see  Figure  6) . 

The  statistics  for  this  drill  (see  Table  3)  indicate  that  students 
were  able  to  recognize  valid  integer  and  real  constants  fairly  well,  though 
they  had  a  little  more  difficulty  recognizing  invalid  constants.   Since  the 
performance  levels  were  quite  acceptable,  no  changes  of  substance  were  made 
to  the  lesson.   The  drill  was  rewritten,  however,  to  shorten  and  simplify 
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Figure  k.        True -False  Drill 
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Figure  5-   Constant  Drill 
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Figure  6.    Penalty  for  an  Early  Exit 
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79  percent  of  the  integer  constants  presented  were  correctly  identified 
91  percent  of  the  real  constants  presented  were  correctly  identified 
56  percent  of  the  invalid  constants  presented  were  correctly  identified 

Mistaken  for  an  integer  constant: 

real  constants:     11 
invalid  constants:   12 

Mistaken  for  a  real  constant: 

integer  constants:   26 
invalid  constants:   56 


Mistaken  for  an  invalid  constant 

integer  constants:   0 
real  constants:      h 


Average  time  for  drill: 
Minimum  time  for  drill: 
Maximum  time  for  drill: 


1U5  seconds 

25  seconds 

286  seconds 


Table  3 


Constant  Identification  Drill  Statistics 


21 


the  code.   In  the  original  version  of  the  drill,  a  new  number  was  randomly 

generated  each  time  a  student  was  asked  to  classify  a  constant.  While  this 

was  a  nice  feature,  it  was  extravagent  of  lesson  space  and  not  worth  the  effort 

since  the  data  gathered  showed  that  practically  all  students  worked  the 

required  ten  problems  correctly  and  immediately  proceeded  on  through  the  lesson. 

The  new  version  of  the  drill  uses  a  random  selection-without-replacement  method, 

selecting  one  of  thirty  constants  from  a  list  of  ten  integer  constants, 

ten  real  constants,  and  ten  invalid  constants. 

The  statistics  on  this  particular  drill  were  skewed  by  the 

behavior  of  one  student.   The  student  eventually  classified  a  grand  total 

of  79  constants;  no  one  else  ever  classified  more  than  20.  Moreover,  he 

classified  71  of  the  79  numbers  he  was  presented  as  real  constants.  His 

behavior  partially  explains  the  high  percentage  of  real  constants  correctly 

identified  and  the  large  number  of  integer  and  invalid  constants  mistaken 
for  reals . 

5 -^-   The  variable  identification  drill 

The  variable  identification  drill  (see  Figure  7)  is  very  similar 
in  form  to  the  constant  identification  drill.   A  student  is  asked  to  classify 
a  given  variable  name  as  integer  in  type,  real  in  type,  or  neither  (invalid 
for  one  reason  or  another) .  As  in  the  constant  identification  drill,  the 
student  must  correctly  work  ten  problems  before  he  can  escape  from  the  drill. 

The  random  selection-without-replacement  method  for  choosing  which 
variable  names  a  student  is  asked  to  classify,  was  first  implemented  in  this 
drill.   It  worked  so  nicely  and  required  such  a  minimal  amount  of  lesson 
space  that  the  constant  identification  drill  was  later  modified  to  use 
the  same  selection  procedure.  From  a  list  of  thirty  variable  names,  ten 
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Figure  7 .   Variable  Drill 


3C)   VARIABLE    DRILL 
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You  might  like  to  try  about  1  J0f  problems 
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integer,  ten  real,  and  ten  invalid,  a  variable  name  is  randomly  selected 
to  be  presented  to  the  student  for  him  to  classify.   Since  the  particular 
variable  name  selected  varies  randomly  from  student  to  student,  it  is 
statistically  highly  improbable  that  students  will  be  presented  the  same 
variable  names  to  classify  in  exactly  the  same  order.   Indeed,  unless 
a  student  works  all  thirty  problems  and  exhausts  the  list  of  variable  names, 
the  subset  of  variable  names  he  will  be  presented  to  classify  will  probably 
be  different  from  the  subset  of  variable  names  another  student  is  presented. 
If  a  student  should  be  ambitious  enough  to  exhaust  the  list  of  variable 
names  by  working  thirty  problems  in  a  row  (data  collection  showed  no 
students  so  inclined),  the  list  of  variable  names  is  restored  and  the  student 
will  begin  reclassifying  variable  names  he  has  already  seen,  though  the 
names  will  appear  in  some  new  random  order. 

The  statistics  for  this  drill  (see  Table  h)    seem  to  indicate 
that  students  recognize  valid  integer  and  real  variable  names  fairly  well, 
but  have  considerably  more  difficulty  discerning  that  illegal  variable 
names  are  indeed  invalid.   Since  the  lesson  text  gives  examples  of  valid 
integer  variable  names  and  valid  real  variable  names,  but  no  examples 
of  invalid  names  perhaps  this  is  to  be  expected.   At  any  rate,  since  the 
lesson  carefully  explains  why  an  invalid  name  is  illegal,  no  changes  were 
made  to  the  lesson  as  a  result  of  this  drill.  The  students'  ability  to 
use  the  given  variable  name  rules  to  identify  valid  names  was  judged  to 
be  adequate  while  the  non-ability  of  students  to  use  the  given  rules  to 
identify  invalid  names  was  thought  to  be  of  little  importance.  After  all, 
any  compiler  given  an  illegal  variable  name  will  produce  error  messages. 
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91  percent  of  the  integer  variable  names  presented  were  correctly  identified 

92  percent  of  the  real  variable  names  presented  were  correctly  identified 

29  percent  of  the  invalid  variable  names  presented  were  correctly  identified 

Mistaken  for  an  integer  variable  name: 

real  variable  names:      3 
invalid  names:  16 


Mistaken  for  a  real  variable  name 

integer  variable  names:   7 
invalid  names:  21 


Mistaken  for  an  invalid  name: 

integer  variable  names:    5 
real  variable  names:      k 


Average  time  for  drill:  110  seconds 
Minimum  time  for  drill:  0  seconds 
Maximum  time  for  drill:     3&0  seconds 


Table  h.        Variable  Identification  Drill  Statistics 
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5  .5  •   The  built-in  function  drill 

The  "built-in  function  drill  (see  Figure  8)  evidently  caused 
students  little  difficulty  so  the  built-in  function  section  of  the  lesson 
was  left  unchanged.  Only  four  wrong  answers  were  recorded  and  most  of  these 
seemed  to  be  due  to  not  understanding  the  difference  between  the  square  of 
a  number  and  the  square  root  of  a  number,  a  subject  not  within  the  scope 
of  this  lesson. 

5 .6 .   Which-order  drill  #1 

The  section  on  operator  precedence  contains  two  exercises  where 
a  student  is  shown  a  FORTRAN  assignment  statement  and  asked  in  "which  order" 
are  the  operations  performed.  The  first  of  these  which-order  drills  was 
so  easy  every  student  answered  the  questions  correctly  (see  Table  5). 

This  being  the  case,  the  exercise  was  judged  to  be  too  simple  and 
a  new  element  of  difficulty  was  added  by  changing  the  statement 
X=R+S**T/U  (the  original  problem)  to  X  =  R  -  SQRT(S)  /  T  **  U 
to  include  a  built-in  function  (see  Figure  9)« 

Average  score:  2  of  2 

Average  time  for  drill:  10  seconds 

Minimum  time  for  drill:  0  seconds 

Maximum  time  for  drill:  22  seconds 

Table  5.   Which-Order  Drill  #1  Statistics 
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Figure  8.   Built-in  Function  Drill 
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Figure   9.       Revised  Which-Order  Drill 
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5. 7-   Which-order  drill  #2 

The  second  which-order  type  drill  also  tested  operator  precedence, 
particularly  for  operators  of  equal  precedence  (see  Figure  10).  When 
statistics  revealed  (see  Table  6)  that  students  had  little  difficulty  with 
the  drill,  the  first  statement  was  made  more  difficult  by  adding  an  unary 
operator.  This  was  done  not  only  to  make  the  drill  more  challenging,  but 
because  other  drills  revealed  students  to  be  weak  on  unary  operators. 

Average  score:  5.2  of  6 

Average  time  for  drill:  39  seconds 

Minimum  time  for  drill:  0  seconds 

Maximum  time  for  drill:  67  seconds 

Table  6.   Which-Order  Drill  #2  Statistics 

5.8.   Multiple  choice  drill 

The  final  drill  of  the  operator  precedence  section  of  fortarith 
is  the  multiple  choice  drill.   In  this  drill,  students  are  shown  a 
mathematical  expression  and  three  FORTRAN  statements.   They  are  then 
asked  which  statement  accurately  calculates  the  mathematical  expression 
shown  (see  Figure  11).   Some  of  the  multiple  choice  questions  gave 
students  a  great  deal  of  trouble  and  this  prompted  a  rethinking  of  both 
the  drill  and  the  preceding  material  (see  Table  7). 

Originally  in  the  drill,  students  were  told  which  answer  was 
correct  and,  if  the  answer  they  gave  was  wrong,  the  reason  why  their 
answer  was  incorrect.   This  all  seems  reasonable  but  one  major  assumption 
is  made.   The  assumption  is  that,  if  the  students'  answers  are  wrong  and  they 
are  told  why  it  is  wrong  and  shown  the  correct  answer,  the  students 


Figure  10.   Revised  Which-Order  Drill  #2 
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Figure  11.   Multiple  Choice  Drill 
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will  immediately  see  why  the  correct  answer  is  correct.   This  does  not 
always  seem  to  be  the  case.   Therefore,  the  drill  was  rewritten  to  tell 
not  only  why  a  student's  incorrect  answer  is  incorrect,  but  why  the 
correct  answer  really  is  correct. 

Data  collection  was  also  useful  in  showing  which  incorrect 
answers  were  favored.  When  writing  a  multiple  choice  drill,  it  is  sometimes 
difficult  to  write  wrong  answers  that  look  vaguely  plausible  rather  than 
totally  absurd.   As  it  turned  out,  students  chose  rather  evenly  between 
"distractors"  in  this  drill,  but  if  they  had  not,  less  favored  incorrect 
answers  could  have  been  replaced  by  equally  wrong  answers  designed  to 
"appear"  more  correct. 

Average  score:  3-8  of  c; 

Average  time  for  drill:  182  seconds 

Minimum  time  for  drill:  kl   seconds 

Maximum  time  for  drill:  368  seconds 

Table  7.   Multiple  Choice  Drill  Statistics 

5-9*    Integer  division  drill 

After  a  section  of  text  describing  the  workings  of  integer 
arithmetic  and  in  particular  integer  division,  students  were  presented  with 
a  drill  consisting  of  five  FORTRAN  statements  to  be  evaluated  (see  Figure  12) 
Originally,  this  drill  had  a  list  of  lettered  answer  choices  and  a  student 
specified  his  answer  by  entering  the  letter  next  to  his  choice.   This  seemed 
unnecessarily  complicated  and  restrictive,  however,  so  the  drill  was  revised 


Figure  12.   Revised  Integer  Division  Drill 
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to  allow  a  student  to  type  in  his  answer  directly.   This  was  the  only 
change  made  to  this  section  of  the  lesson  since  students  really  did  rather 
well  on  the  drill  (see  Table  8). 

Average  score:  k .3   of  5 

Average  time  for  drill:  56  seconds 

Minimum  time  for  drill:  0  seconds 

Maximum  time  for  drill:  195  seconds 

Table  8.    Integer  Division  Drill  Statistics 

5 . 10 .  Mixed  mode  drill 

The  mixed  mode  drill  presents  a  student  with  two  integer  and 
two  real  variables  whose  values  have  already  been  assigned.   The  student 
is  next  shown  an  assignment  statement  with  the  given  variables  combined  in 
an  arithmetic  expression  to  the  right  of  the  equal  sign.   The  student  is 
then  asked  \«rhat  value  will  be  assigned  to  the  variable  left  of  the  equal 
sign  (see  Figure  13) • 

Students  made  quite  a  few  errors  in  this  drill  (see  Table  9) 
particularly  on  one  statement  with  a  unary  minus.   This  led  to  several 
changes  in  the  lesson.   First,  to  give  students  an  additional  encounter 
with  unary  operators,  the  second  "which  order"  drill  was  modified  to  contain 

an  unary  minus.   Secondly,  in  keeping  with  the  idea  that  a  student  should 
fully  understand  why  a  right  answer  is  right,  the  drill's  answer  explanations 
were  modified  to  show  the  intermediate  steps  in  the  evaluation  of  the 
arithmetic  expression.   The  revised  drill  shows  a  student  with  an  incorrect 
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Figure  13 .   Revised  Mixed  Mode  Drill 
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answer  exactly  where  he  went  astray.  Meanwhile,  a  student  who  answered 
correctly  is  able  to  see  graphically  the  step-by-step  procedure  he  went 
through  mentally  to  arrive  at  the  correct  answer.   The  new  answer  explanations 
are  particularly  effective  on  PLATO  since  the  sequence  of  the  intermediate 
results  --  the  fact  that  the  intermediate  calculations  are  performed  one  at 
a  time  in  a  certain  order  --  is  emphasized  by  displaying  the  intermediate 
results  one  at  a  time  in  sequence  on  the  screen. 

Average  score:  2.7  of  U 

Average  time  for  drill:  169  seconds 

Minimum  time  for  drill:  7^  seconds 

Maximum  time  for  drill:  559  seconds 

Table  9.   Mixed  Mode  Drill  Statistics 

5-11.   Time  spent  in  the  different  sections  of  fortarith 

Fortarith  can  be  divided  rather  naturally  into  eight  sections: 
the  title  and  introduction  to  the  lesson,  the  table  of  contents,  and  the 
six  major  topics  taught  in  the  lesson.   Code  was  installed  to  measure  how 
long  a  student  stayed  in  each  area.   The  length  of  time  spent  in  a  particular 
section  (see  Table  10)  turned  out  to  be  roughly  proportional  to  the  amount  of 
code  comprising  the  section.   Not  surprisingly,  a  student  tended  to  spend 
more  time  in  the  sections  with  several  drills  and  less  time  in  the  sections 
with  no  or  only  one  drill . 
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Title  and  introduction 

Table  of  contents 

Assignment  statements 

Constants 

Variables 

Built-in  functions 

Operator  precedence 

Real,  integer,  and  mixed 
mode  arithmetic 


Average 

Minimum 
12 

Maximum 

35 

175 

21 

7 

50 

138 

19 

k-jk 

702 

113 

1,1+85 

228 

7 

668 

111 

6 

328 

568 

272 

901 

553 


12 
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Table  10.  Time  (in  seconds)  Spent  in  the  Different  Sections  of  Fortarith 


5 • 12 .   Help  sequence  usage 

Help  sequences  in  PLATO  are  special  sections  of  code  branched 
to  when  a  student  presses  the  -HELP-  key.  There  are  two  such  sequences  in 
fortarith.  The  first  explains  the  concept  of  scientific  notation  and  is 
available  from  the  section  of  the  lesson  that  introduces  real  constants  in 
E-form.   Scientific  notation  is  mentioned  and  students  are  told  that  a  HELP 
section  is  available  if  they  wish  to  review  exactly  what  scientific 
notation  is. 

The  other  help  sequence  in  fortarith  is  a  "universal"  help  sequence 
that  is  branched  to  when  no  specially  written  help  sequence  is  available. 
It  tells  the  student  no  special  help  sequence  exists  for  the  particular 
section  he  is  currently  studying  and  suggests  that  he  use  the  term  "comment" 
to  leave  a  comment  about  the  lesson  if  he  is  having  difficulty  understanding 
something. 
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Scientific  Notation  Help  Sequence 

Number  of  times  requested:  8 

Number  of  different  students  requesting:       8 

Universal  Help  Sequence 

Number  of  lines  requested:  6 

Number  of  different  students  requesting:       3 

Table  10.  Help  Sequence  Usage  (for  23  students) 

5.13-   Use  of  the  -TERM-  key 

When  a  student  on  PLATO  presses  the  -TERM-  function  key,  he  is 
given  an  arrow  at  which  he  can  enter  an  8-character  "term*   A  lesson  can 
be  easily  programmed  to  recognize  certain  terms  and  take  appropriate  action 

While  the  -TERM-  key  facility  provided  by  PLATO  is  quite  general, 
fortarith  uses  terms  primarily  as  a  jumping  mechanism.   The  terms  "index" 
and  "contents"  return  the  student  to  the  table  of  contents.   The  term 
"notation"  jumps  the  student  to  the  help  sequence  explaining  scientific 
notation.   The  term  "help"  gives  the  universal  help  sequence.   The  term 
"term"  shows  the  list  of  terms  available  in  the  lesson.   The  term  "guide" 
returns  a  student  to  his  or  her  router  lesson,  while  "talk"  and  "comment" 
do  jumpouts  to  lessons  cstalk  and  cscomments,  respectively. 

Students  did  not  use  terms  extensively  but  when  they  did,  most  of 
the  terms  they  typed  in  were  programmed  for  (see  Table  11).   Three  times 
students  entered  terms  that  were  unrecognized.   One  of  these  "scientif , " 
probably  the  8-character  truncation  of  "scientific  notation, "  was  made  into 
an  expected  term  as  a  result  of  the  data  collection.   The  other  two 
unrecognized  terms,  "cs  103"  and  "foot  of"  were  probably  misguided  attempts 
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by  students  to  get  the  list  of  CS  103  lessons  or  to  literally  "go  to  the 
foot  of  the  class." 


Terms  Found 

Term 

Frequency 

index 

k 

guide 

2 

term 

1 

help 

1 

Terms  Not  Found 

Term 

Frequency 

scientif 

1 

cs  103 

1 

foot  of 

1 

Table  11.   Usage  of  the  -TERM-  Key 
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6 .    CONCLUSION 

Data  collection  as  applied  to  fortarith  seems  to  have  been  a 
feasible  and  fruitful  lesson  improvement  technique.   It  -would  seem  to  be  an 
excellent  method  for  improving  any  lesson  of  the  common  "teach  a  little, 
quiz  a  little"  genre.   Whether  the  data  collection  method  would  prove  as 
useful  in  evaluating  a  more  fancifully  written  lesson  is  an  open  question, 
but  one  would  hope  that  with  an  appropriate  approach,  good  results  could 
be  obtained  for  any  lesson. 

Whenever  any  new  technique  is  developed,  some  methods  are  chosen 
for  its  initial  implementation  that  later  prove  to  be  less  than  optimal. 
In  applying  data  collection  to  fortarith  mistakes  were  certainly  made,  most 
minor  but  some  fairly  major. 

The  logistics  of  preparing  the  data  collected  in  the  datafile 
for  analysis  turned  out  to  be  one  of  the  worst  problems.   Obviously,  one 
wants  a  program  to  do  the  analysis  ,  but  PLATO  is  hardly  the  ideal 
system  on  which  to  run  a  large,  time-consuming  analysis  program.   In  the 
first  place,  PLATO  is  designed  for  interaction,  not  number  crunching. 
Secondly,  TUTOR  is  not  the  language  one  would  choose  for  such  a  program. 
Thirdly,  an  analysis  program  of  any  complexity  would  require  a  fairly 
large  amount  of  free  lesson  space,  a  resource  often  not  readily  available. 
With  such  drawbacks  as  these  to  data  analysis  on  PLATO  and  with  an  IBM  360 
handy  across  the  street  at  the  Digital  Computer  Laboratory,  the  idea  of 
doing  the  data  analysis  on  a  computer  other  than  PLATO ' s  CDC  CYBER  70 
sounded  highly  reasonable .  However,  neither  PLATO  nor  TUTOR  offered  any 
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generally  available  facility  by  which  an  author  can  write  the  contents 
of  a  datafile  to  a  tape  or  a  card  punch.  Consequently,  the  author  decided 
to  simply  print  the  contents  of  the  datafile  and  type  up  data  cards  from 
the  listing.   Though  this  sounds  easy  enough,  the  effort  involved  in 
insuring  an  accurate  transcription  of  the  data  came  to  hours  and  hours  of 
excrutiatingly  boring  drudgery.   Pestering  the  PLATO  systems  personnel 
for  some  machine  readable  portable  copy  of  the  datafile  (i.e.,  cards  or  a 
tape)  would  have  taken  time  and  been  trouble,  but  ultimately  would  have 
been  far  less  painful. 

Another  problem  with  data  collection  in  fortarith  was  the 
particular  data  collection  mechanism  installed  in  the  lesson.  Designed 
into  the  data  collection  mechanism  was  the  concept  of  "packing, "  collecting 
several  pieces  of  information  in  a  one  or  two  word  buffer  before  outputting 
the  data  to  the  datafile.  At  first  glance,  packing  would  seem  to  be  a 
thrifty  way  to  conserve  space  in  the  datafile.  However,  if  the  buffer  into 
which  items  are  being  collected  is  partially  full  and  the  student  decides 
not  to  take  the  normal  path  of  the  lesson,  the  data  items  in  the  buffer  are 
lost.   One  solution  to  this  problem,  although  it  is  wasteful  of  datafile 
space,  would  be  to  dispense  with  the  packing  mechanism  entirely  and  output 
each  data  item  immediately  as  it  is  collected. 

With  hindsight,  of  course,  it  is  easy  to  see  things  that  should 
have  been  done  that  were  not.   It  would  have  been  most  interesting  to  record 
the  units  from  which  students  requested  the  universal  help  sequence,  for 
example.   Statistics  on  the  number  of  shift  -NEXT-  and  shift  -BACK-  keypresses 
might  also  have  been  nice.  The  time  statistics,  on  the  other  hand,  were 
most  uninteresting  and  in  future  data  collection  experiments  might  well  be 
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eliminated.   Students  taking  fortarith  were  totally  unaware  their  responses 
were  being  monitored  and  recorded.   In  retrospect,  this  seems  unethical; 
a  student  should  at  least  know  he  is  being  used  as  a  guinea  pig.  Future 
data  collection  experiments  should  begin  with  a  notice  informing  the 
student  that  the  lesson  he  is  about  to  study  contains  code  that  will  monitor 
and  record  his  responses. 

But  enough  about  the  things  the  fortarith  data  collection 
mechanism  failed  to  do  or,  worse  yet,  did  wrong.  What  did  it  do  right? 
It  pointed  the  way  to  specific  changes.  For  example,  in  the  mixed  mode 
drill,  77-8  percent  of  the  students  missed  the  fourth  problem.   On  the 
second  most  difficult  problem  in  that  drill,  only  25. 9  percent  of  the 
students  answered  incorrectly.   The  key  difference  between  the  fourth 
problem  and  the  other  problems  of  the  mixed  mode  drill  was  this:   to  answer 

the  fourth  problem  correctly  a  student  had  to  know  that  an  unary  minus 
has  the  same  operator  precedence  as  addition  and  subtraction.   The  students 
evidently  had  not  picked  that  up  from  the  lesson.  A  special  section  was 
added  to  fortarith  to  specifically  state  that,  in  FORTRAN,  an  unary  plus 
or  minus  has  the  same  operator  precedence  as  a  binary  plus  or  minus.   To 
emphasize  the  point  still  further,  an  earlier  drill  shown  by  the  data 
collection  statistics  to  be  too  easy,  was  modified  to  contain  a  problem 
with  an  unary  minus.   That  is  just  one  example  of  the  sort  of  changes 
data  collection  statistics  generated  throughout  the  lesson. 

In  conclusion  then,  data  collection  seems  to  be  an  effective 
lesson  improvement  technique.   Statistics  gathered  on  students'  use  of  a 
lesson  point  out  a  lesson's  strengths  and,  even  more  clearly,  point  out 
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its  weaknesses.   Once  a  problem  area  is  identified,  the  collected  data 
indicate,  sometimes  with  startling  specificity,  what  changes  need  to  be 
made  to  improve  the  lesson. 


to 
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APPENDIX  A.         SAMPLE   QUESTIONNAIRE 
PLATO  lesson  evaluation  form 


Lesson  name 


Date  taken 


Your  namedf  requested) 


Year  in  school  (circle  one)   Fr.   Soph.   Jr.   Sr.   Grad.   Other(or  nonstudent) 

PLATO  experience( circle  a  or  b) :  a. student  using  PLATO  b. author.  No.  of  lessons  taken 

Among  all  the  lessons  you've  taken,  this  one  would  rate 

Excellent (upper  20%)     good    average     poor     very  poor ( last  20%) 
How  much  time  did  you  spend  in  completing  the  lesson?  minutes 

Please  answer  each  question  below  by  placing  an  X  on  the  space  after  your  response, 
or  on  a  space  between  each  set  of  adjective  phrases  describing  some  part  of  the 
lesson,  amplifying  your  response  whenever  possible  (either  in  the  space  provided, 
or  on  back).   The  continued  improvement  of  PLATO  lessons  depends  on  accurate 
feedback  to  the  author. 


1.  What  was  your  previous  knowledge  of  the  lesson  material.   None  some  — 

2.  Does  the  lesson  fit  in  well  with  others  in  the  sequence? 

yes  somewhat  no  don't  know  


a  lot 


COMMENTS  on  sequence 


Z>.      What  grade  did  the  lesson  give  you? 
COMMENTS  on  grading 


or  none  given 


Examples 


too  many 

very  ?lear 

relevant 


COMMENTS  on  examples 


too  few 
unclear 
irrelevant 


Exercises  for 
the  student 


too  many 
very  helpful 


very  clear 
COMMENTS  on  exercises   


too  few 

worthless 

unclear 


6.   Text  material 

COMMENTS  on  text 


too  much 
very  clear 


too  little 
unclear 


7.   Displ^s  (diagrams,  flow  charts , 
graphs)              excellent 
COMMENTS  on  displays  


poor 


^ 


C.   Organization        well  organized  ™™.i,r 

to        poorly  organized 

too  restrictive  +  rsn   „„„+   * 

COMMENTS  on  organization  t0°  unst™ct^ed 


Total  lesson         learned  a  lot   learned  nothing 

easy  to  use   hard  to  use 

too  much  material  +nr,    -,-+4.-, 

too  little 

much  better  than  a  bonk  „,,  .,, 

much  worse 

much  better  than  a  lecture  v, 

COMMENTS  on  total  lesson  mUCh  W°rSe 


Please  use  the  space  below  and/or  on  the  back  to  describe  any  problems  you  had 
witn  the  lesson.   How  could  the  lesson  be  improved?  List  any  additional  criteria 
which  should  be  used  for  evaluation.  Clonal  criteria 
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